Foreground/background detector

ABSTRACT

A detector for detecting the foreground or background vector near a discontinuity in a vector field has a covering-uncovering detector ( 3 )for determining whether covering or uncovering situation exists. The detector further has calculating means ( 1,2,4 ) for calculating at the position {right arrow over (x)} 1  of the discontinuity:  
     a first position {right arrow over (x)} a  in the previous (covering) or next (uncovering) vector field by shifting {right arrow over (x)} 1  over the first vector at one side of the discontinuity,  
     a second position {right arrow over (x)} b  in the previous (covering) or next (uncovering) vector field by shifting {right arrow over (x)} 1  over the second vector at the other side of the discontinuity  
     and a third intermediate position between {right arrow over (x)} a  and {right arrow over (x)} b . Furthermore, the detector has means ( 5 ) for fetching the background vector with v {right arrow over (a)}v  at the third position in the previous (covering) or next (uncovering) vector field.

[0001] The invention relates to a method for detecting the foreground orbackground vector near a discontinuity in a vector field.

[0002] In several technical fields of imaging technics it is required toknow which velocity around an object edge or velocity discontinuity isthe foreground velocity and which is the background velocity.

[0003] The invention has as an object to provide a method for detectingsaid velocities.

[0004] According to a first strategy of the invention said object isachieved in that it is determined whether covering or uncoveringsituation exists, that at the position {right arrow over (x)}₁ of thediscontinuity

[0005] a first position {right arrow over (x)}_(a) in the previous(covering) or next (uncovering) vector field is calculated by shifting{right arrow over (x)}₁ over the first vector at one side of thediscontinuity,

[0006] a second position {right arrow over (x)}_(b) in the previous(covering) or next (uncovering) vector field is calculated by shifting{right arrow over (x)}₁ over the second vector at the other side of thediscontinuity

[0007] and a third intermediate position between {right arrow over(x)}_(a) and {right arrow over (x)}_(b) is calculated, in which thevector fetched with v_({right arrow over (a)}v) at the third position inthe previous (covering) or next (uncovering) vector field is thebackground vector.

[0008] According to a second strategy of the invention said object isachieved in that it is determined whether covering or uncoveringsituation exists, and that two positions on either side of thediscontinuity are projected to the previous (uncovering) or next(covering) vector field, in which a background velocity is identified asa velocity which crosses the velocity discontinuity and projects to aforeground velocity in the previous picture, whereas a foregroundvelocity projects to itself.

[0009] According to the third strategy of the invention said object isachieved in that it is determined whether covering or uncoveringsituation exists, that the discontinuity is projected to the previousvector field in the covering situation and to the future vector field inthe uncovering situation, near discontinuities it is tested whether thementioned edge has moved over the first vector on one side of thediscontinuity, or over the second vector on the other side of thediscontinuity, and that in case the discontinuity moves with the first(second) vector, the second (first) vector is the background vectorv_({right arrow over (B)}G).

[0010] The invention further relates to a foreground/background detectorhaving corresponding features.

[0011] Further elaborations of the invention are specified in thedependent subclaims.

[0012] The invention will be described in more detail with reference tothe attached drawings.

[0013]FIGS. 1 and 2 illustrate an embodiment according to the inventionof a method for detecting foreground/background in uncovering andcovering respectively;

[0014]FIGS. 3 and 4 illustrate another embodiment according to theinvention of a method for detecting foreground/background in uncoveringand covering respectively;

[0015]FIGS. 5 and 6 show a further embodiment according to the inventionof a method for detecting foreground/background in uncovering andcovering respectively;

[0016]FIG. 7 shows an embodiment of a foreground/background detectoraccording to the invention;

[0017]FIG. 8 shows another embodiment of a foreground/backgrounddetector according to the invention;

[0018]FIG. 9 shows a further embodiment of a foreground/backgrounddetector according to the invention; and

[0019]FIG. 10 shows an image display apparatus according to theinvention.

[0020] The invention can be applied in various technical fields.Examples are:

[0021] picture rate conversion, where it is necessary to determine theforeground and background in a motion estimator (see copendingapplication of the inventor: tritemporal motion estimator). Typicaldevices for this application are TV and PC.

[0022] 3D disparity analysis, where images are generated by a rollingcamera or from a rolling scene, or multiple cameras (at least 3)

[0023] motion based compression where foreground and background regionsare described/compressed differently. An example is compression in MPEG.

[0024] video analysis e.g. for security, where e.g. a foreground personis detected and tracked.

[0025] special effects, where certain foreground or background objectsare extracted, manipulated etc.

[0026] picture enhancement e.g. blurring or deblurring of background.

[0027] In a first strategy shown in FIGS. 1 and 2, the average vectorforeground/background determination, we make use of the fact that anyvectorv_(fetch)=kv_({right arrow over (FG)})+(1−k)v_({right arrow over (B)}G),where k is smaller than 1 and v_({right arrow over (F)}G) andv_({right arrow over (B)}G) are the velocities of the foreground andbackground objects at position {right arrow over (x)}, fetches abackground velocity from the previous vector field in the case ofcovering and a foreground velocity in the case of uncovering. The safestvector to use is the average vector {right arrow over(v)}_(av)=0.5v_({right arrow over (F)}G)+0.5v_({right arrow over (B)}G).More formally, for covering, we calculate the two possible {right arrowover (x)}_(a) and {right arrow over (x)}_(b) of the edge in the previousimage pair, e.g. for covering and a vertical edge at position {rightarrow over (x)}_(E) in vector field n: $\begin{matrix}{{{\overset{->}{x}}_{a} = {{\overset{->}{x}}_{E} + {\overset{->}{D}\left( {{\overset{->}{x} - \begin{pmatrix}1 \\0\end{pmatrix}},n} \right)}}}{{\overset{->}{x}}_{b} = {{\overset{->}{x}}_{E} + {\overset{->}{D}\left( {{\overset{->}{x} + \begin{pmatrix}1 \\0\end{pmatrix}},n} \right)}}}} & (1)\end{matrix}$

[0028] and fetch the vector present at an intermediate position in theprevious vector field (covering) in the ambiguous area: $\begin{matrix}{{{\overset{->}{D}}_{\alpha}\left( {\overset{->}{x},n} \right)} = {\overset{->}{D}\left( {\frac{{\overset{->}{x}}_{\alpha} + {\overset{->}{x}}_{b}}{2},{n - 1}} \right)}} & (2)\end{matrix}$

[0029] If we need to fill in the foreground vector in an ambiguous areaof an interpolation vector field, we choose between {right arrow over(x)}_(a) and {right arrow over (x)}_(b) the one which is most differentfrom {right arrow over (D)}_(a)({right arrow over (x)},n).

[0030] A variant of this first strategy fetches the background vectorfrom the future for uncovering: $\begin{matrix}\left. {{{\overset{->}{D}}_{\alpha}\left( {\overset{->}{x},n} \right)} = {\overset{->}{D}\left( {\frac{{\overset{->}{x}}_{\alpha} + {\overset{->}{x}}_{b}}{2},n} \right)}} \right) & (3) \\{{\overset{->}{x}}_{\alpha} = {{{\overset{->}{x}}_{1} - {{\overset{->}{D}\left( {{\overset{->}{x} - \begin{pmatrix}1 \\0\end{pmatrix}},{n - 1}} \right)}{\overset{->}{x}}_{b}}} = {{\overset{->}{x}}_{1} - {\overset{->}{D}\left( {{\overset{->}{x} + \begin{pmatrix}1 \\0\end{pmatrix}},{n - 1}} \right)}}}} & (4)\end{matrix}$

[0031] with:

[0032] A second strategy (see FIGS. 3 and 4), the twosided self speedforeground/background determination, uses the fact that for uncovering,positions projected to the past with the background velocity have ahigher probability of crossing towards the foreground region than whenthey are projected with {right arrow over (v)}_(av). This is interestingwhen small relative velocitiesv_({right arrow over (F)}G)-v_({right arrow over (B)}G) or inaccuratelyestimated vector fields occur. Because we do not know a priori whichvelocity is the background velocity, we project the two positions oneither side of the edge with its own velocityv_(sel{right arrow over (f)}) (see FIG. 3). As we can see for the idealcase, the lower velocity changes from the background velocity at n tothe foreground velocity at n-1. The probability that a block inforeground in n projects to foreground in n-1 is so high that forpractical reasons we can consider it to be 1. All the other probabledecisions are shown in table 1. TABLE 1 Upper block is in reality:foreground background Decision for upper block FG FG or BG Decision forlower block FG or BG FG

[0033] In case the two projections yield the same (foreground) vector,we have a certain determination. If this vector equals the vector of theupper block, this vector is the foreground velocity vector and viceversa. In case the vectors are different, the method was unsuccessful,and yields an uncertain determination. A similar projection towards thefuture can be applied for the case of covering.

[0034] A third strategy (see FIGS. 5 and 6), the edge projectionforeground/background determination, checks for e.g. uncovering whetherthe edge between v_({right arrow over (F)}G) andv_({right arrow over (B)}G) in the previous image n is present atposition a or b (see FIG. 5). If the edge is detected at position a,$v_{\overset{->}{FG}} = {\overset{->}{D}\left( {{\overset{->}{x} - \begin{pmatrix}1 \\0\end{pmatrix}},{n \pm 1}} \right)}$

[0035] n±1) and vice versa. Care should be taken that the velocities inn-1 are the same velocities as in n, since other velocity edges canoccur in the vicinity of the projection. Obviously again the principlecan be applied by substituting uncovering for covering and the futurefor the past (see FIG. 6).

[0036] It should be noted that the strategies can be enhancedincorporating match errors. In case a crossing to the foreground regionoccurred, the match errors of the vector in that block should be low. Incase we project to a background region that was erroneously allotted aforeground vector in the previous image, the errors should be higher.

[0037] In FIG. 7 an embodiment of a foreground/background detectoraccording to the invention is shown. This detector comprises calculatingmeans 1 and 2 to which the image signal J is supplied for carrying out acalculation at a position {right arrow over (x)}₁ of a velocity edge.The calculating means 1 calculates a first position {right arrow over(x)}_(a) in the previous (covering) or next (uncovering) image byshifting {right arrow over (x)}₁ over the first vector at one side ofthe edge, while the calculating means 2 calculates a second position{right arrow over (x)}_(b) in the previous (covering) or next(uncovering) image by shifting {right arrow over (x)}₁ over the secondvector at the other side of the edge. The choice between previous andnext images is carried out by a covering/uncovering detector 3.

[0038] The outputs of the calculating means 1 and 2 are connected to theinputs of calculating means 4, which calculates a third intermediateposition between {right arrow over (x)}_(a) and {right arrow over(x)}_(b).

[0039] The foreground/background detector of FIG. 7 is further providedwith fetching means 5, one input of which is connected to the output ofmeans 4 for calculating the third position, while to the other input ofwhich the image signal J is supplied. From the output of the fetchingmeans 5 the background velocity V_(BG) could be derived. The fetchingmeans 5 fetches with the velocity at the third position the backgroundvector from the previous (covering) or next (uncovering) image. Thisvector is filled in those regions of the image in the environment of theedge, to which no vector is projected, in case the background vector VPGshould be filled in and the vector chosen between${\overset{->}{D}\left( {{\overset{->}{x} - \begin{pmatrix}1 \\0\end{pmatrix}},n} \right)}\quad {and}\quad {\overset{->}{D}\left( {{\overset{->}{x} + \begin{pmatrix}1 \\0\end{pmatrix}},n} \right)}$

[0040] which is most different from v_({right arrow over (a)}v) isfilled in, in case a foreground vector v_({right arrow over (F)}G)should be filled in. According to a further elaboration the thirdintermediate position is ( {right arrow over (x)}_(a)+{right arrow over(x)}_(b))/2.

[0041] Another embodiment of a foreground/background detector is shownin FIG. 8. This detector comprises projecting means 6, 7 to the input ofwhich an image signal J is supplied. These projecting means 6, 7 arecontrolled by a covering/uncovering detector 8 to which also the imagesignal J is supplied. The control operation of the covering/uncoveringdetection is such that the projecting means 6 projects one position atone side of the velocity edge to the previous (covering) or next(uncovering) image, while the projecting means 7 projects the positionof the other side of the velocity edge to the previous (covering) ornext (uncovering) image. Identification means 10 is connected to theprojecting means 6 and 7, at the output of which the background velocityV_({right arrow over (B)}G) and the foreground velocityV_({right arrow over (F)}G) could be derived. This identification means10 identifies a background velocity as a velocity which crosses thevelocity discontinuity and projects to a foreground velocity in theprevious picture, whereas a foreground velocity projects to itself.

[0042] Preferably a checking means 9 is connected to a projecting means6 and 7, which means checks if the two projections yield the samevector. If so, the identification is certain.

[0043] In FIG. 9 a further embodiment of a foreground/backgrounddetector is shown. This detector is also controlled by acovering/uncovering detector 12. In the detector of FIG. 9 an imagesignal J is supplied to the input of a projecting means 11, which underthe control of the covering/uncovering detector 12 projects thediscontinuity to the previous vector filled in the covering situationand to the future vector field in the uncovering situation. Furthermore,means 13 for determining the velocity on one side of the discontinuityand means 14 for determining the velocity on the other side of thediscontinuity are provided. The determined velocities are supplied totesting means 15 from the output of means 13 and 14. Said testing means15 is connected to the projecting means 11 for testing whether the edgehas moved over the first vector on one side of the discontinuity or overthe second vector on the other side of the discontinuity. The testingmeans 15 delivers the background vector V_({right arrow over (B)}G),which background vector is the second (first) vector in case thediscontinuity moves with the first (second) vector.

[0044] Furthermore filling means could be connected to the testing means20, which filling means fills the first (second) vector in those regionsof the projected vector field in the environment of the discontinuity,to which no vector is projected, in case a foreground vector should befilled in and the other vector is filled in, in case a background vectorshould be filled.

[0045] In FIG. 10 an image display apparatus is shown. This apparatuscomprises a foreground/background detector 16 for detecting aforeground/background vector. An image signal J is supplied to thedetection 16 and also to means 17 for filling the detected foregroundand/or background vectors in images. The detected motion vectors at theoutput of the detector 16 are supplied to the imput of filling means 17.The output of the filling means 17 is applied to a display device 18.The detector 16 could be implemented according to one of above describeddetectors

1. Method for detecting the foreground or background vector near adiscontinuity in a vector field, characterised in that it is determinedwhether covering or uncovering situation exists, that at the position{right arrow over (x)}₁ of the discontinuity a first position {rightarrow over (x)}_(a) in the previous (covering) or next (uncovering)vector field is calculated by shifting {right arrow over (x)}₁ over thefirst vector at one side of the discontinuity, a second position {rightarrow over (x)}_(b) in the previous (covering) or next (uncovering)vector field is calculated by shifting {right arrow over (x)}₁ over thesecond vector at the other side of the discontinuity and a thirdintermediate position between {right arrow over (x)}_(a) and {rightarrow over (x)}_(b) is calculated, in which the vector fetched withv_({right arrow over (a)}v) at the third position in the previous(covering) or next (uncovering) vector field is the background vector.2. Method according to claim 1, characterised in that the intermediateposition is ({right arrow over (x)}_(a)+{right arrow over (x)}_(b))/2.3. Method for detecting the foreground or background vector near adiscontinuity in a vector field, characterised in that it is determinedwhether covering or uncovering situation exists, and that two positionson either side of the discontinuity are projected to the previous(uncovering) or next (covering) vector field, in which a backgroundvelocity is identified as a velocity which crosses the velocitydiscontinuity and projects to a foreground velocity in the previouspicture, whereas a foreground velocity projects to itself.
 4. Methodaccording to claim 3, characterised in that the determination is certainif the two projections yield the same vector.
 5. Method for detectingthe foreground or background vector near a discontinuity in a vectorfield, characterised in that it is determined whether covering oruncovering situation exists, that the discontinuity is projected to theprevious vector field in the covering situation and to the future vectorfield in the uncovering situation, near discontinuities it is testedwhether the mentioned edge has moved over the first vector on one sideof the discontinuity, or over the second vector on the other side of thediscontinuity, and that in case the discontinuity moves with the first(second) vector, the second (first) vector is the background vectorv_({right arrow over (B)}G).
 6. Method according to claim 3, wherein thecrossing from a background region to a foreground region in the previousimage is verified by the match error of the vector in that block. 7.Method according to claim 1, characterised in that the vector fetchedwith v_({right arrow over (a)}v) at the first position in the previous(covering) or next (uncovering) image is filled in those regions of theimage in the environment of the edge, to which no vector is projected,in case the background vector v_({right arrow over (F)}G) should befilled in and the vector chosen between${\overset{->}{D}\left( {{\overset{->}{x} - \begin{pmatrix}1 \\0\end{pmatrix}},n} \right)}\quad {and}\quad {\overset{->}{D}\left( {{\overset{->}{x} + \begin{pmatrix}1 \\0\end{pmatrix}},n} \right)}$

which is most different from v_({right arrow over (a)}v) is filled in,in case a foreground vector v_({right arrow over (F)}G) should be filledin.
 8. Method according to claim 5, characterised in that the mentionededge has moved over the first vector on one side of the edge, or overthe second vector on the other side of the edge, in case the edge moveswith the first vector with which the discontinuity moves, is filled inthose regions of the projected vector field in the environment of thediscontinuity, to which no vector is projected, in case a foregroundvector v_({right arrow over (F)}G) should be filled in, and the othervector is filled in, in case a background vectorv_({right arrow over (B)}G) should be filled.
 9. Detector for detectingthe foreground or background vector near a discontinuity in a vectorfield, characterised by a covering-uncovering detector (3)fordetermining whether covering or uncovering situation exists, calculatingmeans (1, 2, 4) for calculating at the position {right arrow over (x)}₁of the discontinuity a first position {right arrow over (x)}_(a) in theprevious (covering) or next (uncovering) vector field by shifting {rightarrow over (x)}₁ over the first vector at one side of the discontinuity,a second position {right arrow over (x)}_(b) in the previous (covering)or next (uncovering) vector field by shifting {right arrow over (x)}₁over the second vector at the other side of the discontinuity and athird intermediate position between {right arrow over (x)}_(a) and{right arrow over (x)}_(b), and means (5) for fetching the backgroundvector with v_({right arrow over (a)}v) at the third position in theprevious (covering) or next (uncovering) vector field.
 10. Detectoraccording to claim 9, characterised in that the intermediate position is({right arrow over (x)}_(a)+{right arrow over (x)}_(b))/2.
 11. Detectorfor detecting the foreground or background vector near a discontinuityin a vector field, characterised by a covering-uncovering detector (8)for determining whether covering or uncovering situation exists, andmeans (6,7) for projecting two positions on either side of thediscontinuity to the previous (uncovering) or next (covering) vectorfield, in which a background velocity is identified (10) as a velocitywhich crosses the velocity discontinuity and projects to a foregroundvelocity in the previous picture, whereas a foreground velocity projectsto itself.
 12. Detector according to claim 11, characterised in that theidentification is certain if the two projections yield the same vector.13. Detector for detecting the foreground or background vector near adiscontinuity in a vector field, characterised by a covering-uncoveringdetector (12) for determining whether covering or uncovering situationexists, means (11) for projecting the discontinuity to the previousvector field in the covering situation and to the future vector field inthe uncovering situation, and testing means (15) for testing neardiscontinuities whether the mentioned edge has moved over the firstvector on one side of the discontinuity, or over the second vector onthe other side of the discontinuity, in which in case the discontinuitymoves with the first (second) vector, the second (first) vector is thebackground vector v_({right arrow over (B)}G).
 14. Detector according toclaim 11, wherein the crossing from a background region to a foregroundregion in the previous image is verified by the match error of thevector in that block.
 15. Detector according to claim 9, characterisedin that the vector fetched with v_({right arrow over (a)}v) at the firstposition in the previous (covering) or next (uncovering) image is filledin those regions of the image in the environment of the edge, to whichno vector is projected, in case the background vectorv_({right arrow over (F)}G) should be filled in and the vector chosenbetween ${\overset{->}{D}\left( {{\overset{->}{x} - \begin{pmatrix}1 \\0\end{pmatrix}},n} \right)}\quad {and}\quad {\overset{->}{D}\left( {{\overset{->}{x} + \begin{pmatrix}1 \\0\end{pmatrix}},n} \right)}$

which is most different from v_({right arrow over (a)}v) is filled in,in case a foreground vector v_({right arrow over (F)}G) should be filledin.
 16. Detector according to claim 13, characterised in that thementioned edge has moved over the first vector on one side of the edge,or over the second vector on the other side of the edge, in case theedge moves with the first (second) vector with which the discontinuitymoves, the second (first) vector is filled in those regions of theprojected vector field in the environment of the discontinuity, to whichno vector is projected, in case a foreground vectorv_({right arrow over (F)}G) should be filled in, and the other vector isfilled in, in case a background vector v_({right arrow over (B)}G)should be filled.
 17. Image display apparatus comprising aforeground/background detector (16) according to one of the claims 9-16,a filling means (17) for filling the foreground/background vectordetected by the detector (16) and a display device (18) connected to thefilling means (17).